Here is further proof jesus did not have brothers or sister and that in fact the brothers and sisters mentioned in the bible are just relatives.
In Mt. 27:55-56,James and Joses mentioned in Mark 6 as the "brothers" of Jesus, are actually the sons of another Mary. This is clear because this mary is watching jesus crucifixation from far off. The mother of jesus was at the foot of the cross. Another passage to consider is Acts 1:14-15, "[The Apostles] with one accord devoted themselves to prayer, together with the women and Mary the mother of Jesus and with His brothers...the company of persons was in all about a hundred and twenty." A company of 120 persons composed of the Apostles, Mary, the women, and the "brothers" of Jesus. Let's see there were 11 Apostles at the time. Jesus' mother makes 12. The women, probably the same three women mentioned in Matthew 27, but let's say it was maybe a dozen or two, just for argument's sake. So that puts us up to 30 or 40 or so. So that leaves the number of Jesus' brothers at about 80 or 90! Obviously mary did not have this many children.
Lastly it may be seem like joseph would have had sex with his wife based on todays standards. But remember Mary was visited by an angel of god and was told that she would be the mother of the son of god. The bible specifically states that mary did not have sex prior to the birth of jesus. If they did not have sex while they were pregnant, what is the difference, why have sex after. It makes sense that joseph may have considered that mary was purified by god and he may have respected her special relationship with god and not had sex with her.
Hope this helps.